Storage Systems and Methods for Robotic Picking

ABSTRACT

A robotic system including a storage structure provided with vertical members supporting a grid collectively defining grid spaces and a plurality of storage containers arranged in vertical stacks for storing inventory items, each one of the vertical stacks configured to be arranged within a respective one of the grid spaces. One or more load handling robots are installed upon the grid and include a lifting device extendable in a vertical direction to extract one or more storage containers from the vertical stacks and transport the extracted storage container to a station. The robotic system further includes one or more manipulator robots installed at the station and within a single grid space. The one or more manipulator robots include a picking arm provided with a pneumatic gripping tool arranged to pick an inventory item from the transported storage container and place the picked item into an order container.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. application Ser. No.16/856,409, filed Apr. 23, 2020, which is a continuation of U.S.application Ser. No. 16/831,963, filed Mar. 27, 2020, which is acontinuation of U.S. application Ser. No. 16/804,251, filed Feb. 28,2020, which claims the benefit of the filing date of U.S. ProvisionalPatent Application No. 62/961,390, filed Jan. 15, 2020 and the benefitof the filing date of U.S. Provisional Patent Application No.62/879,843, filed Jul. 29, 2019, the disclosures of which are eachhereby incorporated herein by reference.

BACKGROUND OF THE DISCLOSURE

The present disclosure generally relates to storage systems andinventory retrieval methods, and more particularly, to a storage systemand a mobile, manipulator robot for retrieving inventory items from thestorage system.

Warehouses, or distribution fulfillment centers, require systems thatenable the efficient storage and retrieval of a large number of diverseproducts. Traditionally, inventory items are stored in containers andarranged on rows of shelving on either side of an aisle. Each container,or bin, holds a plurality of items of one or more product types. Theaisles provide access between the shelving for an operator or robot tomigrate the aisles and retrieve the items. It is well understood thatthe aisles reduce the storage density of the system. In other words, theamount of space actually used for the storage of products (e.g., theshelving) is relatively small compared to the amount of space requiredfor the storage system as a whole. As warehouse space is often scarceand expensive, alternative storage systems that maximize storage spaceare desired.

In one alternative approach, which offers a significant improvement instorage density, containers are stacked on top of one another andarranged in adjacent rows. That is, no aisle is provided between theadjacent rows of stacked containers. Thus, more containers, and in turninventory, can be stored in a given space.

Various methods for retrieving inventory from the stacked containershave been contemplated. U.S. Pat. No. 10,189,641, for example, disclosesa system in which containers are stacked and arranged in a plurality ofrows underneath a grid. Vehicles equipped with a lifting apparatusnavigate the grid and lift a desired container. The container is thentransported down a port to a picking/sorting zone, where an operator orrobot picks individual products from the container and sorts theproducts into one or more order containers. To minimize unnecessarytransportation of the containers, each container is typicallytransported to the picking/sorting zone only after multiple orders of aspecific product have been received.

Despite the increased storage density provided by the known stackedstorage system, various shortcoming remain. For example, orderfulfilment times are often lengthy, particularly for products that areordered infrequently because the containers are retrieved in priority asa function of the number of products of one type that have been ordered.Additionally, the vehicles are required to navigate long distances(which takes considerable time and consumes considerable battery power)while driving bins back-and-fourth to the transportation ports.Furthermore, the required picking/sorting zones reduce the overallstorage density of the warehouse and add additional complexity andcosts. While the throughput of the stacked storage system can beincreased by adding additional vehicles to the grid (or by modifying thesystem to include additional container transportation ports), there is alimit to the amount of vehicles that can be operated on the grid beforethe grid becomes overly congested with vehicles and the throughput ofthe system declines due to gridlock.

BRIEF SUMMARY OF THE DISCLOSURE

In accordance with a first aspect of the present disclosure, a highdensity storage structure is provided. The storage structure includessupport members configured to house a plurality of containers, a firstset of parallel rails to support a mobile, manipulator robot and a fluidsupply line having a plurality of valves disposed within the supplyline. Each of the valves have a closed condition in which the supplyline is in fluid isolation from an outside environment and an opencondition in which the supply line is in fluid communication with theoutside environment such that a mobile, manipulator robot traversing thefirst set of parallel rail may receive a fluid supply from the fluidsupply line when the valve is in the open condition.

In accordance with another aspect of the disclosure, a mobile,manipulator robot for retrieving inventory from the storage structure isprovided. The robot may include a body having an interface configured tosend processor readable data to a central processor and receiveprocessor executable instructions from the central processor, a mobilityassembly coupled to the body, a coupler selectively mateable to a portto receive a fluid supply from a supply line, and a picking armconnected to the body. The picking arm may be coupled to a firstpneumatic gripping tool configured to pick inventory items.

In accordance with yet another aspect of the disclosure, a method ofcontrolling a mobile, manipulator robot to retrieve a product storedwithin a container located in a storage structure is provided. Themethod may include moving the mobile, manipulator robot over a first setof parallel rails of the storage structure and to a picking location,identifying a grasping region located on a product based at least inpart upon image data obtained by a sensor attached to the mobile,manipulator robot, adjusting a picking arm equipped with a pneumaticgripping tool to a grasping pose, and grasping the product using thepneumatic gripping tool.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic perspective view of a frame structure for housinga plurality of stacked containers in a known storage system.

FIG. 2 is a schematic plan view of a portion of the frame structure ofFIG. 1 .

FIGS. 3A and 3B are schematic perspective views, from the rear and frontrespectively, of a known load handler device for use with the framestructure depicted in FIGS. 1 and 2 .

FIG. 3C is a schematic perspective view depicting a container beinglifted by the load handler device of FIGS. 3A and 3B.

FIG. 4 is a schematic perspective view of the frame structure of FIG. 1having a plurality of the load handler devices of FIGS. 3A-3C installedon the frame structure.

FIG. 5 is a schematic perspective view of the storage system of FIG. 4depicting a digging operation to retrieve a target container from astack of containers.

FIG. 6A is a schematic illustration of a robotic system including astorage structure for housing a plurality of stacked containersaccording to an embodiment of the present disclosure.

FIG. 6B is a schematic perspective view of the storage structure of FIG.6A.

FIG. 6C is a schematic perspective view of two storage structuresarranged on top of one another according to another embodiment of thepresent disclosure.

FIG. 6D is a schematic side elevation view of a digging robot performinga digging operation within the storage structure of FIG. 6B.

FIG. 7A is a perspective view of a rail depicting a channel extendingthrough the rail and a conduit extending from the channel to a surfaceof the rail.

FIG. 7B is an enlarged view of a portion of the rail of FIG. 7A.

FIG. 8A is a cross-section view of a valve disposed within the conduitof the rail of FIGS. 7A and 7B.

FIG. 8B is an enlarged view of a portion of the rail of FIG. 8A.

FIG. 9A is a schematic perspective view of a manipulator robot includinga picking arm equipped with a pneumatic gripping tool and a tool holder,installed on top of the storage structure of FIG. 6B.

FIG. 9B is a schematic enlarged view of a portion of the robot of FIG.9A.

FIG. 9C is a flow chart showing an example method of determining agrasping pose of the picking arm of the robot of FIG. 9A.

FIGS. 9D and 9E are schematic illustrations of grasping regions ofproduct items stored within a container.

FIG. 9F is a perspective view of a plurality of pneumatic gripping toolsbeing stored in the tool holder of FIG. 9A.

FIG. 10 is a top elevation view illustrating a mobility assembly of therobot of FIG. 9A.

FIG. 11 is a schematic cross-section view of a coupler of the robot ofFIG. 9A.

FIG. 12A is a perspective view of the picking arm of the robot of FIG.9A.

FIG. 12B is a side view of a portion of the picking arm of FIG. 12A.

FIG. 12C is a schematic cross-section view of a target containerillustrating the picking arm of FIG. 12A picking inventory items ofdifferent sizes from the target container.

FIG. 13 is a cross-section view illustrating the coupling between thepneumatic gripping tool of FIG. 9A and the picking arm of FIG. 12A.

FIGS. 14A and 14B are schematic cross-sections illustrating the couplingbetween the coupler of FIG. 11 and the conduit of FIGS. 7A and 7B.

FIG. 15 is a flow chart showing a method of grasping a product itemusing the picking arm and the pneumatic gripping tool of FIG. 13 .

FIG. 16 is schematic illustration of an alternative pneumatic system foruse with the manipulator robot of FIG. 9A.

FIG. 17 is a cross-section view of a modified gripping tool for use withthe alternative pneumatic system of FIG. 16 .

FIG. 18 is a partial perspective view of a modified storage structureincluding a gantry frame supporting a robotic picking arm equipped witha pneumatic gripping tool.

FIG. 19 is a schematic illustration of another modified storagestructure including an assembly positioned above the rails and fluidsupply line extending from the assembly toward the rails.

DETAILED DESCRIPTION

As used herein, when terms of orientation, for example, “vertical” and“horizontal” or relative terms such as, “above,” “upwards,” “beneath,”“downwards” and the like are used to describe the orientation orrelative position of specific features of the storage structure ormanipulator robot, the terms are in reference to the orientation or therelative position of the features in the normal gravitational frame ofreference when the storage structure is positioned with a bottom of thestorage structure resting on a surface. Also as used herein, the terms“substantially,” “generally,” and “about” are intended to mean thatslight deviations from absolute are included within the scope of theterm so modified.

FIGS. 1 and 2 illustrate a frame structure for efficiently storing aplurality of stackable containers 10, also known as bins, within astorage system according to the prior art. Containers 10 are stacked ontop of one another to form stacks 12 and are arranged in a framestructure 14. Each bin 10 typically holds a plurality of product items(not shown). The product items within each bin 10 may be identical, ormay be of different product types.

Frame structure 14 includes a plurality of vertical members 16 thatsupport a first set of parallel horizontal members 18 extending in afirst direction (e.g., the X-direction), and a second set of parallelhorizontal members 20 extending in a second direction (e.g., theY-direction). Horizontal members 18 and horizontal members 20 form aplurality of horizontal grid spaces within which stacks 12 are housed.Frame structure 14 is thus constructed to guard against horizontalmovement of the stacks 12 of bins 10, and to guide vertical movement ofthe bins.

The uppermost level of frame structure 14 includes rails 22 arranged ina grid pattern across the top of horizontal members 18 and horizontalmembers 20. With additional reference to FIGS. 3A-3C and 4 , rails 22support a plurality of robotic load handling devices 30. A first set ofparallel rails 22 a guides movement of load handling devices 30 in afirst direction (e.g., the X-direction) across the top of framestructure 14, and a second set of parallel rails 22 b, arrangedperpendicular to the first set of parallel rails, guides movement of theload handling devices in a second direction (e.g., the Y-direction)across the top of the frame structure. In this manner, rails 22 allowload handling devices 30 to move laterally in two directions (in theX-direction and in the Y-direction) across the top of frame structure14, so that the load handling devices can be moved into position aboveany one of the stacks 12 of bins 10.

Each load handling device 30 includes a vehicle 32 with a first set ofwheels 34, consisting of a pair of wheels on the front of the vehicleand a pair of wheels on the back of the vehicle, arranged to engage withtwo adjacent rails of the first set of rails 22 a. Similarly, a secondset of wheels 36, consisting of a pair of wheels on each lateral side ofthe vehicle, is arranged to engage with two adjacent rails of the secondset of rails 22 b. Each set of wheels 34, 36 can be lifted and lowered,so that either the first set of wheels 34 or the second set of wheels 36is engaged with the respective set of rails 22 a, 22 b depending on thedesired direction of movement of vehicle 32.

When the first set of wheels 34 is engaged with the first set of rails22 a and the second set of wheels 36 is lifted clear from the second setof rails 22 b, the first set of wheels can be driven, by way of a drivemechanism (not shown) housed in vehicle 32, to move the load handlingdevice 30 in the X-direction. To move the load handling device 30 in theY-direction, the first set of wheels 34 is lifted clear of rails 22 a,and the second set of wheels 36 is lowered into engagement with thesecond set of rails 22 b. A drive mechanism (not shown) associated withthe second set of wheels 36 can then be used to drive the second set ofwheels in the Y-direction.

Load handling device 30 is also equipped with a crane device 40 having acantilever arm 42 that extends laterally from the top of vehicle 32. Agripper plate 44 is suspended from cantilever arm 42 by cables 46 thatare connected to a winding mechanism (not shown) housed within vehicle32. Cables 46 thus can be spooled into or out from cantilever arm 42 toadjust gripper plate 44 with respect to the vehicle 32 in theZ-direction.

Gripper plate 44 is adapted to engage with the top of a bin 10. Forexample, gripper plate 44 may include pins (not shown) that mate withcorresponding holes (not shown) in the rim that forms the top surface ofbin 10, and sliding clips (not shown) that are engageable with the rimto grip the bin. The clips are driven into engagement with bin 10 by asuitable drive mechanism housed within gripper plate 44, which may bepowered and controlled by signals carried through cables 46, or througha separate control cable (not shown).

To remove a bin 10 from the top of a stack 12, the load handling device30 is moved as necessary in the X and Y directions so that the gripperplate 44 is positioned above the stack in which the desired bin islocated. Gripper plate 44 is then lowered and brought into engagementwith the bin 10 on top of stack 12, as shown in FIG. 3C. After the clipshave engaged with and secured to bin 10, gripper plate 44, and in turnthe bin, may then be pulled upwards by spooling cables 46. At the peakof its vertical travel, bin 10 is accommodated beneath cantilever arm 42and is held above rails 22. In this way, load handling device 30 cantransport bin 10 to another location. Cables 46 are long enough to allowhandling device 30 to retrieve, and place, bins 10 at any depth withinstack 12, including the floor level. Vehicle 32 is sufficiently heavy tocounterbalance the weight of bin 10 and to remain stable during thelifting process. Much of the weight of vehicle 32 is attributed to thelarge and heavy batteries that are required to power and operate thedrive mechanisms of wheels 34, 36.

The known storage system, as shown in FIG. 4 , may include a pluralityof load handling devices 30 that operate simultaneously to increase thethroughput of the system. The system depicted in FIG. 4 includes twoports 24, or shafts, for transferring bins 10 into or out of the system.An additional conveyor system (not shown) may be associated with eachport 24. In this manner, bins 10 that are transported to port 24 by loadhandling device 30 can be subsequently transferred to a picking/sortingstation (not shown) where the products contained in the bins are pickedand sorted into individual orders. Similarly, bins 10 can be moved bythe conveyor system to port 24 from an external location, such as abin-filling station (not shown), and transported to a stack 12 by theload handling devices 30 to restock the system.

If it is necessary to retrieve a bin (“target bin”) that is not locatedon the top of stack 12, then the overlying bins 10 a (“non-target bins”)(e.g., the bins located between the target bin 10 b and rails 22) mustfirst be moved to allow load handling device 30 to access the targetbin. This operation is referred to as “digging”.

FIG. 5 illustrates a known digging operation in which one of the loadhandling devices 30 sequentially lifts each non-target bin 10 a from thestack 12 of bins 10 containing target bin 10 b. Each of the non-targetbins 10 a may be placed in a temporary location on top of another stack12. After each of the non-target bins 10 a have been removed, target bin10 b can be extracted from frame 14 by load handling device 30 andtransported to port 24. After target bin 10 b has been extracted,non-target bins 10 a may be placed back in the original stack 12 torestore the original order of the stack less the target bin.

Each of the load handling devices 30 may be operated under the controlof a central computer. Each individual bin 10 in the system is tracked,so that the appropriate bins can be retrieved, transported and replacedas necessary. For example, during a digging operation, the temporarylocations of each of the non-target bins 10 a is logged, so that thenon-target bins can be replaced in the stack in a particular order.

While the storage system illustrated in FIGS. 1-5 allows for the densestorage of products, it requires the transportation of entire containersof products back-and-forth between the stacks and the picking/sortingzones, during which time products cannot be picked and sorted into newincoming orders, thus reducing total system throughput. In order tominimize bin transportation, target bins 10 b are typically onlyretrieved and transported to the picking/sorting stations after multipleorders have been placed for a product item of one type. Although thismethod reduces bin transportation, order fulfilment times are oftenlengthier than desired, particularly if an order contains one or moreproducts that are infrequently ordered by consumers. For this reason,“piece picking” inventory from the known frame structure 14 has beencontemplated. U.S. Pat. Pub. Nos. 2018/0319590 and 2018/0346243, forexample, disclose a robot equipped with a picking arm to pick individualitems from a container located in frame structure 14. Nevertheless, thepicking robots and systems disclosed in U.S. Pat. Pub. Nos. 2018/0319590and 2018/0346243 are not robust enough to handle the picking of a widevariety of products.

The present disclosure, on the other hand, provides a robot having apicking arm equipped with a pneumatic end-effector (e.g., gripping tool)to grasp a variety of products and to place the products directly intoone of a plurality of order containers. To date, a major barrier indeveloping robotic picking arms has been the inability of the pickingarm to consistently grasp products of varying sizes, shapes, weights,materials, surface textures, densities, mass distributions, stiffnessesand fragilities. While picking arms equipped with pneumatic grippingtools have been contemplated as one potential solution for gripping awide variety of products, these gripping tools require extensive suctionforce and flow rate that can only be produced by large vacuum pumpsand/or compressors (e.g., smaller vacuum pumps/compressors are onlycapable of providing adequate suction for a very small range of items).Oversized pneumatic compressors and/or vacuum pumps, however, areprohibitively large for load handling device 30 or similarly sizedvehicles. In other words, load handling device 30 is not capable ofcarrying a large pneumatic compressor and/or vacuum pump onboard whilenavigating the rails 22 of frame structure 14, and modifying the loadhandling device to carry the oversized pneumatic compressor and/orvacuum pump would require a substantially larger vehicle body 32 suchthat the footprint of the load handling device would consume a largenumber of grid spaces. As a result, fewer load handling devices would beable occupy the grid at a single time, and throughput of the systemwould be reduced. For this reason, manipulator robots with pneumaticgripping tools have generally been confined to the floor of a warehouseand are often fixed to a stationary base.

The present disclosure provides a robotic system including a storagestructure equipped with a pneumatic air supply system and a compact,manipulator robot with one or more pneumatic gripping tools selectivelycoupleable to the pneumatic air supply system. As a result, the robotcan grasp a large variety of products while traversing across thestorage structure and support larger payloads during grasping. Theability of the manipulator robot to quickly and efficiently grasp a widevariety of inventory items is further improved by the robots ability toquickly switch between two or more pneumatic gripping tools and requestgrasping assistance from a teleoperator, if the robot is unable toautonomously grasp an item during an edge case scenario (or thepredicted control instructions have high uncertainty or low confidence)such that the manipulator robot can continue its normal operation withminimal downtime or interruption. These improvements, among otheradvantages, are discussed in further detail in this disclosure.

FIG. 6A is a schematic illustration of a robotic system 100 according toan embodiment of the present disclosure. A robot, such as manipulatorrobot 200, may be housed in a warehouse 101, or other fulfillmentcenter, and tasked with picking inventory items contained within storagestructure 114. Robot 200 may operate in one of two modes: an autonomousmode, by executing autonomous control instructions, or a tele-operatedmode, in which the control instructions are manually piloted (e.g.,directly controlled) by an operator. While the term “controlinstructions” (whether autonomous or piloted) is primarily describedherein as instructions for grasping an item, it will be appreciated thatthe term may additionally refer to a variety of other robotic tasks suchas the recognition of an inventory item, the placement or release of agrasped item (e.g., in a particular location or orientation) or anyother robotic task that facilitates order fulfillment. In oneembodiment, robot 200 may be a machine learning robot capable ofexecuting autonomous or piloted control instructions.

Robotic system 100 includes one or more teleoperator interfaces 102, atleast one of which may be located at a remote site outside of warehouse101, one or more processor-based computer systems 103, each of which arecommunicatively coupled via one or more network or non-networkcommunication channels 104, and one or more storage devices 105, whichstores, for example, a machine learning grasp pose prediction algorithmused to predict new grasping poses for manipulator robot 200 to executeand grasp inventory items. While storage device 105 is illustrated asbeing separate from computer system 103, in at least someimplementations, the storage devices can be an integral part orcomponent of the computer system (e.g., memory such as RAM, ROM, FLASH,registers; hard disk drives, solid state drives).

Operator interface 102 includes one or more input devices to capturecontrol instructions from an operator and one or more output devices.The one or more user interface devices 102 may be, for example, apersonal computer, a tablet, (smart) phone, a wearable computer, and thelike. Exemplary input devices include keyboards, mice, touch screendisplays, displays (e.g., LCD or OLED screen), controllers, joysticksand the like. Exemplary output devices include, without limitation,displays (e.g., LCD or OLED screen), head mounted displays, speakers,and/or haptic feedback controllers (e.g., vibration element,piezo-electric actuator, rumble, kinesthetic, rumble motor). Operatorinterface 102 thus may be utilized by an operator to observe roboticpicking, for example, aspects of manipulator robot 200 and/or theinventory stored within storage structure 114. Operator(s) may view orsee a representation of manipulator robot 200 performing one or moretasks such as grasping an item by reviewing one or more still and/ormoving images of the manipulator robot and/or its environment. Theseimages and/or video may be replayed and/or viewed in real time. Ifmanipulator robot 200 is unsuccessful at autonomously performing thetask, the operator can utilize operator interface 102 and instruct therobot to perform one or more robotic tasks such as grasping a productitem and/or releasing the product item into a desired order container.Although operator interface 102 is primarily designed to assist robot200 in performing tasks that the robot is struggling to perform, such asgrasping, it will be appreciated that a teleoperator can utilize theoperator interface at any time (including prior to a failed graspingattempt) to manually control the robot to perform any manipulation taskand/or override autonomous control instructions.

Computer system 103 coordinates the operation of robotic system 100.Computer system 103 can be a processor based computer system. Theprocessor may be any logic processing unit, such as one or moremicroprocessors, central processing units (CPUs), digital signalprocessors (DSPs), graphics processing units (GPUs),application-specific integrated circuits (ASICs), programmable gatearrays (PGAs), programmed logic units (PLUs), and the like. In someimplementations, computer system 103 may include a control subsystemincluding at least one processor. Computer system 103, the at least oneprocessor and/or the control subsystem may be interchangeably referredto herein as the processor, the controller, the central computer, thecomputer, the server or the analyzer.

Examples of a suitable network or non-network communication channels 104include a wire based network or non-network communication channels,optical based network or non-network communication channels, wireless(i.e., radio and/or microwave frequency) network or non-networkcommunication channels, or a combination of wired, optical, and/orwireless networks or non-network communication channels.

Mobile, manipulator robot 200 includes an interface to send and/orreceive processor readable data or processor executable instructions viacommunication channels 104 to computer 103. In this manner, computer 103can predict grasping poses (e.g., position and/or orientation and/orposture of the robotic picking arm) based on inventory item data (e.g.,the geometry and material of an item and its specified pose) and sendcontrol instructions to manipulator robot 200 to execute the predictedgrasping pose and grasp the product item. If the control instructionsare unsuccessful in performing a task (e.g., grasping the item), or thecentral computer determines that the predicted control instructions areunlikely to be successful, the system can automatically requestintervention from the operator, allowing robot 200 to be teleoperativelycontrolled from a local or remote location.

As will be described in greater detail hereinafter, the present systemallows a teleoperator to remotely pilot manipulator robot 200 and movethe robot into a variety of grasping (or manipulation) poses to trainthe machine learning system to more accurately predict future autonomousrobot control instructions.

Although FIG. 6A illustrates two robots 200 located within a singlewarehouse 101, it will be appreciated that the system can include asingle robot or any number of robots located within a single warehouse,or one or more robots located within a plurality of warehouses. Therobotic system is thus advantageously configured to allow one or moreoperators to teleoperatively pilot or control a plurality of manipulatorrobots 200, via one or more operator interfaces 102, from a site locatedlocal or remote to the warehouses in which the robots are contained.

Storage structure 114, as shown in FIG. 6B, is configured to efficientlystore stackable containers 110, also referred to as bins. The containers110 are stacked on top of one another to form stacks 112. Each bin 110is configured to hold a plurality of product items (not shown) which maybe identical, or of different product types. Containers 110 preferablyhave an open end through which the products can be retrieved. The openend of container 110 may be an open top end or an open lateral side. Thebottom of containers 110 may be inwardly tapered to facilitate therolling and/or the sliding of inventory products toward the center ofthe container and away from the sidewalls to facilitate picking, and insome cases may include slidable, pivotable or bomb bay doors to dump theinventory items into other containers or elsewhere.

Storage structure 114 includes vertical members 116 that support a firstset of horizontal members 118 extending in a first direction (e.g., theX-direction), and a second set of horizontal members 120 extending in asecond direction (e.g., the Y-direction). Horizontal members 118 andhorizontal members 120 form a plurality of horizontal spaces for housingstacks 112. The horizontal spaces are constructed to guard againstlateral movement of the stacks of bins 110. Storage structure 114 mayadditionally include one or more ports 121 or shafts to transfer binsinto or out of the storage structure. A conveyor belt or shuttle system(not shown) may be associated with each port 121 to transport bins 110to an external location. For example, a bin containing products forshipment may be transported down port 121 to an external location forfurther packaging and/or shipment, while an empty bin may be transporteddown the port to a bin-filling station (not shown) for replenishment,and then subsequently transported up the port and to one of the stacks112 to restock the storage structure.

The uppermost level of storage structure 114 may include a first set ofrails 122 extending in a first direction (e.g., X-direction), and/or asecond set of rails 124 extending in a second direction (e.g.,Y-direction). In embodiments in which storage structure 114 includes thefirst set of rails 122 and the second set of rails 124, the combinationof the first and second set of rails forms a horizontally oriented grid126 having a plurality of grid spaces 127. Rails 122, 124 allow one ormore robots to move about the grid 126 above the stacks 112 of bins 110.At least one of the vertical members 116, horizontal members 118,horizontal members 120 or rails 122, 124 may define a channel thattransports fluid such as compressed air to the robots installed on grid126 as is discussed in further detail hereinafter.

As shown in FIG. 6C, a plurality of similarly constructed storagestructures 114 with shallower stacks (e.g., fewer containers per stack)may be layered on top of one another to reduce the time it takes to diga target container 110 b (e.g., the container storing a desiredproduct), which in turn, increases the throughput of the system. In suchscenarios, each storage structure 114, or level, would be spaced apartfrom an adjacent level with enough clearance between each level to allowone or more robots to move about a respective grid 126. One or moreelevators and/or ramps having inclined and/or declined rails 129 (in theZ-direction) may be provided between the grids 126 of adjacent storagestructures 114 to allow the robots to migrate between the levels asdesired.

Referring back to FIG. 6B, one or more of the lateral sides of storagestructure 114 may additionally or alternatively include the second setof rails 124 extending in the second direction (e.g., Y-direction),and/or a third set of rails 125 extending in a third direction (e.g.,Z-direction). In embodiments in which storage structure 114 includes thesecond set of rails 124 and the third set of rails 125, the combinationof the second and third set of rails 124, 125 forms a verticallyoriented grid 126 having a plurality of grid spaces 127. Manipulatorrobot 200 may traverse vertical grid 126, extract bins 110, and pickfrom the extracted bins housed in shelving, racks or stacks on thelateral sides of storage structure 114. When the term “grid” is usedherein without an orientation qualifier (e.g., vertical or horizontal),the term may refer to any grid structure formed by a combination ofrails 122, 124, 125, whether the grid be horizontally oriented orvertically oriented.

It is also envisioned that a plurality of similarly constructed storagestructures 114 may be positioned laterally adjacent to one another (notshown), to increase storage capacity. In such scenarios, each storagestructure 114 would be spaced apart from an adjacent storage structurewith enough space between the adjacent storage structures to allow arobot 200 to traverse about a respective vertically oriented grid 126and access containers 110 housed within either of the adjacent storagestructures.

Referring to FIGS. 7A and 7B, each one of the rails 122, 124, 125forming grid 126 may be extruded or otherwise formed from a highlyconductive metal such as aluminum. A power source P may be coupled togrid 126 to supply a voltage to rails 122, 124, 125 and, in turn, toselectively provide a voltage to robot 200 to recharge small batteriesor super/ultra-capacitors of the robot and/or directly power the variousdrive mechanisms of the robot. The power may be transferred from grid126 to robot 200 in one of several methods. For example, grid 126 mayhave a single polarity such as a negative charge, while a structure orceiling above the grid (not shown) is positively charged (or viceversa). In this embodiment, robots 200 may include an antenna whichcontacts the positively charged structure or ceiling above grid 126 andcompletes the circuit between the opposite polarities. In an alternativearrangement, adjacent rails of one set of the parallel rails 122,parallel rails 124 and/or parallel rails 125 may have oppositepolarities such that when robot 200 is disposed on the adjacent parallelrails, the wheels, or conductive brushes (e.g., contact elements), ofthe robot will complete the circuit. For example, a first one of theparallel rails 122 may have a positive polarity while an adjacent one ofthe parallel rails 122 may have a negative polarity. In this manner,robot 200 need not include the large onboard batteries associated withload handling device 30. As a result, robot 200 is less bulky and moremaneuverable than its load handling device 30 counterpart.

Rails 122, 124, 125 may include a double u-channel or profiled trackhaving an upper surface 128, outer surfaces 130, inner surfaces 132 anddrive surfaces 136 a, 136 b (collectively “drive surfaces 136”). In thismanner, two robots may traverse a single rail 122, 124, 125, increasingthe number of robots capable of driving on grid 126 at any given time.For example, a first robot supported on drive surface 136 a may pass asecond robot supported by drive surface 136 b. The upper surface 128,outer surfaces 130 and inner surfaces 132 of rails 122, 124, 125 may beanodized or painted with a non-conductive coating to prevent the robotor storage structure 114 from short circuiting and to minimize the riskof electrocution. In other words, the inner drive surfaces 136 of rails122, 124, 125 may be the only surfaces of the rails that remain at leastpartially or entirely electrically charged (aside from the terminalends, or a small section of the terminal ends of the rails, which arenot anodized for the purpose of transmitting power along the rails ofthe grid).

Storage structure 114 further includes a fluid supply system 138configured to supply fluid such as compressed air to robot 200 when therobot is installed on rails 122, 124, 125. Fluid supply system 138 thuseliminates the need for robot 200 to carry a bulky onboard aircompressor or vacuum generator to grasp inventory items using itspneumatic gripping tool 248 (FIG. 9A). Fluid supply system 138 includesa fluid source S and a supply line 140. Fluid source S may be acompressor, such as a pneumatic compressor, to supply compressed air tosupply line 140. Alternatively, fluid source S may be a vacuum pump orvacuum generator.

While supply line 140 is primarily described and illustrated herein asextending through the rails 122, 124, 125 of grid 126, it will beappreciated that the supply line may alternatively be formed by orextend at least partially through the channels of vertical members 116,horizontal members 118 or horizontal members 120 forming the frame ofstorage structure 114, be attached to or otherwise coupled to anexternal surface of the rails and/or the frame structure, or otherwisebe in close proximity of the rails so long as the fluid supply isaccessible to manipulator robot 200 when the robot is positioned on thegrid.

As shown in FIG. 7A, supply line 140 may include a series of channels142, conduits 144 and ports 146. Channels 142 may extend along an entirelength of rails 122, 124, 125, and are preferably, embedded within alower portion of the u-channel such that the channels extendcontinuously in a longitudinal direction of a respective rail withoutinterruption at the intersections of rails 122 and rails 124, or theintersection of rails 124 and 125. A plurality of conduits 144 mayextend between channel 142 and a port 146 located at a surface of arespect rail. In a preferred embodiment, at least one of rails 122, 124,125 that surrounds each one of grid spaces 127 has a conduit 144. Grid126 is thus capable of supplying fluid such as compressed air to robot200 irrespective of the robot's location on the grid.

Referring to FIGS. 8A and 8B, a plurality of valves 150 may be disposedwithin supply line 140, for example, within the channels formed byvertical members 116, horizontal members 118, horizontal member 120, orwithin the channels 142 of the conduits 144 of rails 122, 124, 125. Eachvalve 150 is transitionable between a closed condition in which thecompressed air is contained within supply line 140, and an opencondition in which the supply line is in fluid communication with theenvironment such that compressed air may be supplied to manipulatorrobot 200. Each valve 150 may include a biasing member 152, such as aspring, and a plug 154 coupled to the spring to seal port 146. Whenspring 152 is in a neutral or unbiased condition, the spring 152 biasesthe plug into the port 146, which seals the compressed air within supplyline 140. Alternative valves may be used to seal compressed air withinsupply line 140. For example, the valve may be constructed as anypassively or actively actuated valve capable of being transitionedbetween a closed condition and an open condition, such as anelectrohydraulic servo valve.

With specific reference to FIG. 8B, the rails 122, 124, 125 of grid 126may define a cavity 143 aligned with a longitudinal axis of conduit 144.Cavity 143 may include a tapered edge extending from the upper surface128 of rails 122, 124, 125 toward port 146. A magnet 157, or otherferrous material, may surround port 146 to magnetically couple robot 200to grid 126 during the transference of the compressed air from supplyline 140 to robot 200. A gasket, such as an O-ring 155, may be providedaround port 146 to seal the connection between robot 200 and grid 126,and/or at any other location surrounding the valves 150 to preventcompressed air from leaking out of supply line 140.

The compressed air of supply system 138 may be selectively accessed bymobile, manipulator robot 200, shown in FIGS. 9A and 9B, to provide thenecessary suction to allow the manipulator robot to piece-pick inventoryitems ranging in sizes, shapes, weights, materials, surface textures,densities, mass distributions, stiffnesses and fragilities.

Referring to FIGS. 9A and 9B, manipulator robot 200 includes a vehiclebody 202, a mobility assembly 204 configured to guide movement of thevehicle body along rails 122, 124, 125 and a picking arm 206.Manipulator robot 200 also includes a communication interface to sendand receive data between the manipulator robot and central computer 103and/or the manipulator robot and teleoperator interface 102. The datamay include information relating to the position of manipulator robot200 to enable central computer 103 to control movement of the robotabout grid 126 or about warehouse 101 in general. The data may alsoinclude sensor data relating to or providing Product Information (e.g.,location, dimensions, shapes, weights, materials, porosities, surfacetextures, colors, densities, mass distributions, stiffnesses,fragilities or the like) that assist the computer or a teleoperator indistinguishing between different products located in the containerand/or predicting a grasping pose for grasping the product item.

Vehicle body 202 may be formed of four sidewalls 208 a, 208 b, 208 c,208 d (collectively “sidewalls 208”), an open bottom end 210 and an opentop end 212. The sidewalls 208 are preferably sized such that vehiclebody 202 has a footprint of a single grid space 127. In other words,when robot 200 is positioned on the horizontal grid 126, two opposingsidewalls (e.g., 208 a, 208 c) are positioned over two adjacent rails122 extending in the X-direction, while the other two opposing sidewalls(e.g., 208 b, 208 d) are positioned over two adjacent rails 124extending in the Y-direction. In other embodiments, the vehicle body 202of robot 200 may have a footprint that is larger than a single gridspace 127. The open bottom end 210 and the open top end 212 of vehiclebody 202 allow picking arm 206 to extend through the vehicle body andgrasp a product contained in a target bin 110 b, which may be locateddirectly beneath the body (e.g., the bin located on the top of the stackof bins aligned with the vehicle body in the Z-direction). Picking arm206 may alternatively be used to pick products contained in target binslocated laterally adjacent to the vehicle body 202 as shown in FIG. 9A.

One or more of the sidewalls 208 of vehicle body 202 may optionallyinclude a pivotable digging plate (not shown) for digging into a stack112 and pulling a target bin to the top of a particular stack and/or fortransporting bins for replenishment purposes. The digging plate may bepivotable between a collapsed condition in which the digging plate liesflush against a respective interior or exterior surface of the sidewall208 of vehicle body 202, and an operating condition in which the diggingplate extends radially away from and perpendicular to the respectivesidewall of the vehicle body. The digging plate may be similar togripper plate 44 of load handling device in that the digging plate isconfigured to be lowered in the Z-direction and brought into engagementwith any of the bins 110 located in stack 112. Like gripper plate 44,the digging plate may be adapted to pull bins 110 upwards by spoolingcables, which are long enough to retrieve a target bin located at anydepth within stack 112. However, robot 200 need not include a diggingplate or another mechanism for digging the containers from stack 112.System 100 could instead rely on the combination of manipulator robot200 and a separate robot specifically adapted to perform digging tasks.The digging robot may be known load handling device 30 or digging robot205 (FIGS. 6C and 6D). With specific reference to FIG. 6D, digging robot205 may include a vehicle body having a container receiving cavity and adigger 207 extendable beneath the body. Digger 207 may be a scissor liftor include a series of telescoping beams or other compact linearactuators with long stroke. In this manner, digger 207 may reach beneathgrid 126 to lift a single container 110, or a plurality of containers(e.g., a target bin and each of the non-target bins overlying the targetbin), through the receiving cavity and above the grid in a single lift.Alternatively, digger 207 may be positioned on a single external side ofdigging robot 200 and include a latching device such as a hook forengaging with a lateral side of containers 110. In this manner, diggingrobot 205 may reach beneath grid 126 to lift a single container 110, ora plurality of containers (e.g., a target bin and each of the non-targetbins overlying the target bin) above the grid and on a lateral side ofthe digging robot (e.g., without lifting the containers through thedigging robot). The digger 207 of digging robot 205 may be electrically,pneumatically or otherwise actuated.

The internal surface of the sidewalls 208 of robot 200 may also includea latch, hook, plate or other mechanism (not shown) for coupling orderbins 214 a, 214 b (collectively “order bins 214”) within the vehiclebody 202 of the manipulator robot such that the combination of the piecepicking robot and the one or more order bins have a footprint ofapproximately one grid space 127. The latch, hook, digging plate orother mechanism may alternatively be placed on an external surface ofone or more of the sidewalls 208 of vehicle body 202 to couple orderbins 214 around the vehicle body as shown in FIGS. 9A and 9B.

Each of the order bins 214 may correspond to one or more orders. If asingle order bin corresponds to more than one order, the bin may bepartitioned to separate the multiple orders in a single bin, or remainun-partitioned with all of the items from multiple orders mixedtogether. For example, order bin 214 a may correspond to a firstconsumer's order and order bin 214 b may correspond to a secondconsumer's order. Thus, after robot 200 has picked a product from targetbin 110 b, the product may be placed directly into the order bincorresponding to the order of the consumer who purchased the product. Inone embodiment, the bottom end of order bins 214 may include slidable,pivotable or bomb bay doors to facilitate the dumping of items intoother containers, areas, or down ports 121 for further sorting orprocessing. It will be appreciated, however, that piece picking robot200 need not carry any order bins 214. Instead, piece picking robot 200may be used only for grasping products, which may be subsequently placedinto order bins 214 carried by a “transporting robot” (e.g., a robotwhose primary responsibility is carrying around order bins) (not shown).In this manner, both manipulator robot 200 and the transporting robotmay move along grid 126 and meet at certain picking or transferlocations.

With specific reference to FIG. 9B, robot 200 further includes one ormore sensors 262 such as a camera, video recorder, Light Detection andRanging (LIDAR), and the like, oriented to capture pictures, pointclouds, video etc. (generally referred to herein as “an image” or“images”) of the product item(s) stored within containers 110. Theimage(s) may then be transmitted via network or non-networkcommunication channels 104 to processor 103 which, in some instances,may additionally be relayed to operator interface 102. In this manner,processor 103 may implicitly or explicitly analyze the images and thenexecute a machine learning algorithm, located on storage device 105 topredict a grasping pose to grasp the desired product item, beforetransmitting the grasping pose control instructions to robot 200 viacommunication channels 104 which, when executed by the robot, causes thepicking arm 206 of the robot to approach and attempt to grasp the item.Although the grasping pose can refer to a single pose, grasping an itemoften requires a set of consecutively run poses. As used herein, theterm ‘grasping pose’ may refer to a single pose or a set ofconsecutively run poses. The images are preferably continuously capturedas robot 200 traverses grid 126 and transmitted to central computer 103.In this manner, central computer may determine a grasping pose for thepicking arm 206 of robot 200, or the picking arm of another manipulatorrobot, before the manipulator robot reaches the picking position, thusincreasing throughput of robotic system 100.

FIG. 9C is a flow chart showing a method 400 of autonomously determininga grasping pose. The process for determining a grasping pose may begin,at 402, with a command from processor 103 that instructs sensors 262 tocapture an image of the inventory disposed within a target container 110b.

The image(s) may then be transmitted, at 404, over network ornon-network communication channels 104 to processor 103. Upon receipt ofthe image, processor 103 may analyze the images and the ProductInformation of the items stored within the target container 110 b.

Based on the Product Information, processor 103 may execute one or moregrasping pose detection algorithms (which can be neural networks ormachine learning algorithms stored on storage device 105) to predict oneor more grasping pose candidates. Processor 103 may then implement apolicy, at 410, which utilizes one or more metrics, checks and filtersto select one or more of the predicted grasping pose candidates forrobot 200 to execute sequentially or to add to its queue. Then, at 412,processor 103 produces, makes, or generates a signal including processorreadable information that represents the selected grasping pose andsends the signal through communication channels 104 to robot 200. Itwill be appreciated, however, that robot 200 can alternatively run partof, or the entirety of, the grasping model on an onboard computer ratherthan relying on remote computing and communications.

As shown in FIGS. 9D and 9E, sensors 262 and the grasping model may workin concert to identify a grasping region 414 of product items, definedas a specific area on the product item or packaging of the product itemas a whole that manipulator robot 200 has a high likelihood ofsuccessfully grasping. Grasping region 414 may be a relativelynon-porous and flat surfaced region of the product item and/or theproduct packaging. FIG. 9D illustrates product items of different typeswithin a target container 110 b. FIG. 9E illustrates the identificationof a grasping region 414 of the product items located within an area ofthe target container.

Referring to FIG. 10 , mobility assembly 204 is configured to guidemovement of vehicle body 202 along rails 122, 124, 125 and positionrobot 200 over, or laterally adjacent to a target bin 110 b (e.g., a bincontaining the product to be picked). Mobility assembly 204 may includea plurality of wheels 216, a motor 218 and one or more transmissions(belts or linkages) 220 operably coupling each one of the wheels to themotor. Each one of the wheels may include a direct drive (not shown), ahub motor (not shown), a gear drive actuator (not shown) or a belt driveactuator (not shown) to rotate wheels 216 and move vehicle body 202along the rails in which the wheels are positioned. Mobility assembly204 may include four wheels 216, with one wheel being located at oradjacent to each one of the corners of vehicle body 202. The orientationof wheels 216 is controlled by motor 218 and transmission 220. Morespecifically, transmission 220 couples motor 218 to each one of wheels216 such that rotation of the motor simultaneously rotates/pivots theorientation of each one of the wheels 216 between a first orientation inwhich each of the wheels are oriented, for example, along rail 122, anda second orientation in which the wheels are aligned with rail 124(i.e., 90 degrees). The four wheels 216 can thus be used to guidemovement of vehicle body 202 in two directions, for example, along rails122 (e.g., X-direction) and along rails 124 (e.g., Y-direction).Consequently, robot 200 need not include a second set of wheels or aseparate drive mechanism for lifting and disengaging the second set ofwheels each time the robot drives along a different rail, as is the casewith known load handling device 30.

The mobility assembly 204, or body 202, of manipulator robot 200 mayfurther include one or more electrical brushes or conductive elements221 (shown in FIG. 14B) to engage the inner drive surfaces 136 a, 136 bof grid 126 and transfer the charge from the grid to a relatively smallonboard battery or super/ultra-capacitor, and in turn, to the drivemotor or gear drive actuator. As a result, robot 200 may charge itsbattery or super/ultra-capacitor while robot 200 traverses grid 126. Thethroughput of the system is thus increased because robot 200 need not beremoved from grid 126 and/or paused in order to charge or swap itsbattery or super/ultra-capacitor. The relatively small onboard batteryor super/ultra-capacitor also allows robot 200 to be lighter, faster andsafer than its load handling device 30 counterpart. Moreover, the smallbattery or super/ultra-capacitor may temporarily power the drive motorand/or gear drive actuator to drive the wheels even when robot 200 isremoved from and driven off of grid 126. Robot 200 may be driven, forexample, on the warehouse floor in any direction to navigate the robotbetween grids 126 and/or to other areas of the warehouse to assists withother fulfillment tasks such as replenishment, picking/sorting at apicking/sorting station and/or packaging.

Referring to FIG. 11 , robot 200 further includes a pneumatic coupler222 adapted to receive a fluid, such as compressed air, from supplysystem 138. Coupler 222 is attached to the vehicle body 202 of robot 200in a manner that allows the coupler to selectively engage and disengagewith valve 150. For example, the coupler may be lowered toward grid 126to engage valve 150 and raised away from the grid to disengage thevalve. Coupler 222 may be a generally hollow tube sized to be positionedwithin the cavity 143 of rails 122, 124, 125. The mating end of coupler222 may be tapered and/or include a self-alignment or misalignmenthandling device to assist in positioning coupler 222 within cavity 143.The mating end of coupler 222 may also include an O-ring (not shown), amagnet 223 for magnetically engaging the magnet 157 or ferrous materialdisposed around port 146, and a device 224 for transitioning valve 150between the closed and open conditions. Device 224 may be, for example,a mechanical member adapted to push plug 154 into conduit 144 (away fromport 146), or any other device for electrically, magnetically,mechanically or otherwise transitioning valve 150, or another valve,between the closed and open conditions. For example, a similarlyconstructed coupler may include one or more conductive pads to providepower and actuate an electrohydraulic servo valve.

Robot 200 may optionally carry a small air tank 266 (FIG. 9A) forstoring compressed air. In some embodiments, air tank 266 may be smallerthan 20 cubic feet. The air tank 266 of robot 200 is in selectivecommunication with coupler 222. In this manner, robot 200 need notaccess the compressed air of supply system 138 each time the robotdesires to grasp a product item. Robot 200 may instead rely on thecompressed air stored within air tank 266 to pick inventory items for alimited time, and need only couple to supply system 138 when the robotdesires to refill the air tank. As a result, robot 200 can temporarilyoperate picking arm 206 on the grid without coupling to supply system138 and/or when the robot is driven off of the grid to assist with othergrasping and sorting tasks.

FIGS. 12A and 12B illustrate an example embodiment of picking arm 206coupled to pneumatic gripping tool 248. Picking arm 206 is moveable withseveral degrees of freedom to position pneumatic gripping tool 248relative to inventory stored in any position within a container 110 andhas long stroke (in the Z-direction) to allow robot 200 to lift anysized item from the container, and to deposit the item in order bin 214.In the illustrative embodiment, picking arm 206 may include a basemember 226, one or more horizontal extensions 228, a vertical extension230 and a positioning arm 232 configured to removably secure pneumaticgripping tool 248. Positioning arm 232 may be a relatively thin tubethat has a smaller diameter than gripping tool 248. This allowspositioning arm 232 to freely position gripping tool 248 withincontainer 110 without interference from other items or partitionsdisposed within the container. Positioning arm 232 may be coupled tovertical extension 230 via a coupling mechanism 233 that allows thepositioning arm to translate along a “first linear pathway,” such as atrack, extending along the length of the vertical extension. One or morefluid lines are disposed within positioning arm 232 to fluidly couplegripping tool 248 and coupler 222 (FIG. 11 ). If more than one fluidline is utilized, the fluid lines may be independent from one another.

Base member 226 may be attached to one of the sidewalls 208 and extendabove the open top end 212 of vehicle body 202. Base member 226 mayinclude a “second linear pathway” 227, such as a track, extending thelength of the base member. Horizontal extensions 228 may be coupled tobase member 226 and vertical extension 230 in a manner that allows thehorizontal members to move along the second linear pathways 227, tovertically position pneumatic gripping tool 248 relative to theinventory items. Horizontal extensions 228 are also rotationally coupledto the base member, one another, and the vertical extension via joints236 that allow the pneumatic gripping tool to be positioned relative tothe product items with several degrees of freedom. As will be furtherexplained with reference to FIG. 12C, in a preferred embodiment, thecombination of the first and second linear pathways is equal to orgreater than 2 times the height of containers 110, and preferably equalto or greater than 3 times the height of the containers.

FIG. 12C is a schematic, cross-section view of target container 110 bholding a first, relatively small item 238 (in height) and a second,relatively large item 240 (in height) that is approximately the heightof the container. The long stroke picking arm 206 of robot 200 iscapable of picking either of the items 238, 240 from target container110 b and depositing the items in order bin 214. For example, in pickingitem 238 from the bottom of target container 110 b, gripping tool 248must first be lowered a distance equal to the height of the robot body(shown in FIG. 9A) and the height of target container 110 b (e.g., adistance equal to approximately 2 times the height of the targetcontainer). Thus, in order to pick item 238, horizontal extensions 228may be moved to the bottom of first linear pathway 227 and positioningarm 232 may be moved downward relative to vertical extension 230, and tothe bottom of the second linear pathway, to allow gripping tool 248 tocontact and grasp the relatively small item 238. After item 238 has beengrasped, positioning arm 232 may be moved upward, and to the top of thefirst linear pathway, and the horizontal extensions may be moved upwardsalong the second linear pathway 227, allowing the first item to bedeposited within order bin 214.

On the other hand, while picking arm 206 need not be lowered deep intotarget container 110 b to grasp the relatively large second item 240, inorder to deposit the second item in order bin 214, the gripping toolmust be raised to a sufficient height that allows the bottom of thesecond item to clear the top of the order bin (e.g., gripping tool 248must be positioned a distance equal to approximately the height of thecontainer above the top of the order bin). Thus, after the relativelylarge second item 240 has been grasped by picking arm 206, positioningarm 232 may be retracted upwards relative to vertical member 230 alongthe first linear pathway and the horizontal members may move toward thetop of second linear pathway 227 to allow the bottom of the second item240 to clear the top of order bin 214. Thus, it will be appreciated thatin order to grasp and deposit relatively flat items such as item 238 andrelatively tall items such as 240, the stroke of picking arm 206 (in theZ-direction) must be at least 2 times the height of the containers, andpreferably 3 times the height. While the stroke length may beaccomplished with a single linear pathway, dividing the stroke lengthinto a plurality of linear pathways allows picking arm 206 to be morecompact and have a smaller vertical profile.

In a preferred embodiment, as shown in FIG. 12B, picking arm 206 alsoincludes a spring 257 or a compliant gripping element (and/or aback-drivable actuator, or a force controlled actuator that exhibitsactive compliance and functions as a virtual spring) that providespassive or active compliance. The spring 257 (and/or back-drivableactuator or force controlled actuator) may be provided between pneumaticgripping tool 248 and positioning arm 232, and/or at the couplingmechanism that couples positioning arm 232 and vertical extension 230.Thus, if gripping tool 248 presses against a product or infrastructureof the storage structure with too great a force, the gripping tool orthe positioning arm 232 will recoil to prevent damage to the picking arm206 and/or the product. The compliance may also better position grippingtool 248 relative to the item during grasping.

It will be understood that picking arm 206 may be alternativelyconstructed and/or include fewer or additional components, so long asthe pneumatic gripping tool is positionable with several degrees offreedom to grasp inventory items stored within target container 110 b.For example, picking arm 206 may also include a load cell to measure thepayload of a grasped item and/or sense an external force applied to thegripping tool. In this manner, robot 200 can instantaneously determineand/or verify the identity of the grasped item.

As mentioned above, gripping tool 248 is in fluid communication withcoupler 222 and thus in selective communication with fluid source S. Inembodiments in which fluid source S is a pneumatic compressor providingcompressed air, robot 200 may include one or more air ejectors, airaspirators, Venturi pumps 244 (FIG. 11 ) or similar devices capable ofusing the compressed air of supply system 138 to produce a vacuum orsuction force (hereinafter “Venturi pump”). A “bypass valve” may beprovided between coupler 222 and Venturi pump 244. The bypass valve istransitionable between 3 conditions: a closed condition, a first opencondition and a second open condition. In the closed condition, thebypass valve prevents the compressed air from passing to gripping tool248. In the first open condition, the bypass valve allows compressed airto flow from coupler 222, through Venturi pump 244 and to gripping tool248. Thus, when the bypass valve is in the first open condition, thevalve allows compressed air to flow through Venturi pump 244 such thatthe Venturi pump can generate a suction force to operate a gripping tool248 that relies on suction for grasping. In the second open condition,the bypass valve allows compressed air to flow from coupler 222 togripping tool 248 but diverts the compressed air around Venturi pump244. Thus, when the bypass valve is in the second open condition, thecompressed air bypasses the Venturi pump and allows robot 200 to utilizethe compressed air to actuate a pneumatic gripping tool 248 such asclamp and/or the additional tool elements discussed below. Thecompressed air may also be utilized to blow or dispel air from grippingtool 248 to reposition inventory items within containers 110 and/or toreposition inventory items within order bins 214 to facilitate packing.Additional valves (“variable valves”) may be provided upstream of thebypass valve (e.g., between coupler 222 and the “bypass valve”) toprecisely regulate airflow to the bypass valve and, in turn, to grippingtool 248.

Referring to FIG. 13 , positioning arm 232 includes a magnet 246, forexample, a ring magnet 246 or a magnet arrangement that magneticallycouples gripping tool 248 to the positioning arm. Gripping tool 248 maybe any pneumatically actuated tool for grasping items. For example,pneumatic gripping tool 248 may be a suction cup having a sidewall 251formed of a resilient material such as rubber with bellows 250, and agroove 249 positioned above the bellows. The sidewall 251 of grippingtool 248 is thus adapted to compress when the gripping tool engages anobject. Gripping tool 248 may further include a lip 252 formed from aresilient material, which also may be a rubber, such that the lip of thegripping tool is adapted to deform to and create a seal with the surfaceof a product in which it engages. A ring magnet 254 may be provided ongripping tool 248 to attract the magnet 246 of positioning arm 232 andto magnetically couple the gripping tool to the positioning arm. Agasket, such as an O-ring 256, may be provided on gripping tool 248 toseal the connection between positioning arm 232 and the gripping tool.In some embodiments, gripping tool 248 may additionally have a groove(not shown) to cooperate with a protrusion (not shown) on thepositioning arm 232 to prevent rotational and axial movement of thegripping tool relative to the positioning arm when the gripping tool iscoupled to the positioning arm. In other embodiments, gripping tool 248may be coupled to positioning arm 232 via any other non-magneticquick-change mechanism such as a push/pull connection or a twist-lockedconnection.

Gripping tool 248 may additionally include one or more tool elements toassist in gripping or other order fulfillment processes. For example,gripping tool 248 may include a clamp (not shown) having a plurality ofpneumatically actuated fingers for grasping the target product. Thefingers may be used in combination with the suction cup or in isolationof the suction cup. In some embodiments, the fingers themselves mayinclude suction cups. In other embodiments, gripping tool 248 mayinclude an array of suction cups on a single gripping tool. A singlegripping tool 248 may, for example, include a plurality of suction cupsarranged in an array to grip large and heavy inventory item at severaldiscrete locations, thereby providing a more stable grasp than a singlesuction cup, or to grasp multiple items at once. In further embodiments,the additional tool elements may include other gripping elements such asuniversal jamming grippers, foam vacuum grippers, pneumaticallyinflatable fingers, pressure actuated fingers, pneumatically actuatedlinkage or piston driven grippers with rigid or compliant fingers or anyother pneumatically driven or vacuum driven (positive or negativepressure) gripper elements. Gripping tool 248 can also includeconductive target pads and push-pins on the gripping side of thegripping tool (or vice-versa) to provide power and communication signalsto internal sensors and/or actuators of the gripping tool or toelectrically supplement the pneumatic grasp. In other embodiments, thetool element(s) may be formed as any pneumatically actuated tool andneed not include a “gripping” element for grasping items. The toolelement may be, for example, a knife for cutting open boxes. Theadditional tool elements may be provided on gripping tool 248 or as aseparate tool. In embodiments in which a plurality of gripping elementsare utilized on a single gripping tool 248, or the gripping toolincludes an additional tool elements, each gripping element or toolelement may be coupled to an independent and discrete fluid line suchthat each gripping element or tool element may be independentlyactuated. Each fluid line may have a Venturi pump 244, bypass valve andone or more variable valves associated with the fluid line to controlthe suction force or the force of compressed air as explained above.

The combination of pneumatic gripping tools 248 and any one or more ofthe additional tool elements may be used to grasp one or more objects ata time, manipulate objects within order bins 214, pack grasped objects,swap battery packs on the robot, activate bomb bay doors on a bin, liftand attach an order bin to a container, cut or seal boxes, manipulateitems within an order bin, for example, by nudging, blowing or topplingthe items, or perform any other operations that facilitate orderfulfillment.

Referring to FIGS. 9A and 9F, the vehicle body 202 of robot 200 mayinclude a tool holder 258 for holding a plurality of gripping tools 248.Tool holder 258 may include a plurality of retainers 260 a, 260 b(collectively “retainers 260”) such as arcuate or rectangular cutoutsfor receiving the groove 249 of gripping tool 248, or a holding areasuch as a cup for receiving the sidewall 251 of the gripping tool. Inthis manner, a plurality of different gripping tools 248 (e.g., grippingtools having different tool elements and/or number of or configurationsof tool elements, or suction cups having lips of different sizes,materials, shapes, configurations or orientations) may beinterchangeably coupled to positioning arm 232, and to tool holder 258when not in use, such that the picking arm 206 can select a particulargripping tool based upon the size, shape, material or weight of theproduct in which the robot is tasked with grasping. In some embodiments,tool holder 258 may be magnetic to assist in securing gripping tools248. Tool holder 258 may alternatively, or additionally, include acompliant member to secure gripping tools 248 within retainers 260 via asnap-fit connection. Each of gripping tools 248 may include an RFID, ARtag, calibrated weight or similar identifier capable of being identifiedby a sensor, such as sensors, 262 or the load cells of picking arm 206.In this manner, robot 200 can determine if a gripping tool is secured topicking arm 206 and can verify that the secured gripping tool is thedesired gripping tool.

Referring back to FIG. 9B, one or more sensors 264, such as a scanner,may be positioned on the vehicle body 202 or the picking arm 206 ofrobot 200 to scan picked products and determine and/or verify whichorder bin 214 the picked product should be deposited. The scanning fieldof the scanners may be multiplied by positioning mirrors on the innersurfaces of sidewalls 208. Sensors 264 or another sensor may be used tocapture an image or data of a grasped product after it has been picked(and before it has been deposited in order bins 214) to identify and/ordetermine the size and dimensions of the item. This information can betransmitted to central computer 103 which can then instruct manipulatorrobot 200 to deposit the grasped item within a particular location ofone of order bins 214 and/or in a particular orientation to facilitatedense packing. In some embodiments, vehicle body 202 may also include aledge upon which a grasped item may be temporarily placed, andsubsequently re-grasped from a different orientation, in order to adjustthe grasp to facilitate packing.

Use of robotic system 100 to piece pick individual product items fromcontainers 110 will now be described. To begin, robot 200 may use itspicking arm 206 to grasp one or more order bins 214 and attach the binsto its own vehicle body 202 or the vehicle body of another robot.Alternatively, order bins 214 may be attached to robot 200 by thedigging plate or another device on the robot, or external to the robot,or with the assistance of an operator. Robot 200 may then beautonomously positioned on grid 126 and operated under the control of acentral computer 103, which continuously logs the location of each ofthe robots, containers 110 and products contained within the containers.Central computer 103 is additionally designed to efficiently control themovement of robots 200.

When one or more orders are received, the computer assigns the orders toone or more of the manipulator robots 200 based upon the current ordervolumes of each of the robots and the locations of the productscontained in the order. If the product is located beneath a non-targetbin, robot 200, or a separate digging robot 205 located nearby, may pulltarget bin 110 b to the top of stack 112. For example, digging robot 205may position itself over a stack 112 containing the target bin 110 b.Digging robot 205 may then extend digger 207 underneath the diggingrobot and between vertical members 116 and stack 112 (on a single orboth lateral sides of the stack) to grasp the target bin 110 b and eachof the non-target bins 110 a positioned between the target bin and grid126. Each of the grasped bins may then be lifted such that thenon-target bins are lifted, for example, through the receiving cavity ofthe digging robot and the target bin is positioned within the receivingcavity. Digging robot 205 may then drive to a location over a separatestack 112 that is missing a single container, and release target bin 110b on the top of that stack such that robot 200 can pick items from thetarget bin. In releasing target bin 110 b, digging robot 205 may releaseonly the target bin (e.g., never release non-target bins 110 a) orrelease the target bin and the non-target bins to stack the non-targetbins on top of the target container such that the bottom most non-targetbin is positioned within the receiving cavity of the digging robot andthe other non-target containers are stacked above the receivingcontainer of the digging robot, before and again securing all of thenon-target bins, driving back to the original stack and depositing thenon-target bins in the original stack, and in the original order, lessthe target bin.

With target bin 110 b at the top of stack 112, the central computer thenautonomously directs the assigned robot 200 to a first position on grid126 located above or adjacent to a first one of the products containedin the order. Mobility assembly 204 allows robot 200 to navigate rails122, 124, 125 and move to the desired position on grid 126. Robot 200may then transition valve 150 to its open condition to receive pneumaticair to pick from target bin 110 b.

More specifically, as is shown in FIG. 14A, robot 200 may be coupled togrid 126 by positioning coupler 222 within cavity 143 such that themagnet 223 of the coupler engages the magnet 157 surrounding port 146.Insertion of coupler 222 into cavity 143 may be aided by the taperededges of the coupler and the tapered edges of the cavity. In thismanner, if coupler 222 is slightly misaligned with respect to port 146,the tapered edge of the coupler will slide down the tapered edge of thecavity to guide the coupler into proper alignment with the port.

Alignment may also be aided by the magnetic connection between themagnet 223 of coupler 222 and the magnet 157 or ferrous materialsurrounding port 146. The magnetic connection also aids in securingcoupler 222 within cavity 143 against the upwardly directed force ofcompressed air that is created as device 224 compresses plug 154 intoconduit 144 (away from the upper surface 128 of rails 122, 124, 125)while valve 150 is transitioned to the open configuration, therebyallowing pneumatic air to flow around the plug and into the coupler. Inthe event that the valve is an electrohydraulic servo valve, the couplermay be similarly engaged with the valve such that the conductive targetpads provide power to electrically transition the valve from the closedcondition to the open condition. The electrohydraulic valve mayalternatively be transitioned by a voltage received from grid 126 uponreceiving a signal from robot 200 or central computer 103.

With fluid supply system 138 coupled to robot 200, the robot mayimmediately use the compressed air for grasping and/or store thecompressed air within air tank 266 for later use. In embodiments inwhich pneumatic gripping tool 248 relies on a suction force to graspobjects, the one or more Venturi pumps 244 can use the compressed airprovided by pneumatic air source S to generate a suction force foroperating gripping tool 248.

Upon arrival at a desired grid space 127, the picking arm 206 andpneumatic gripping tool 248 may immediately be positioned in thegrasping pose as instructed by the central computer 103, as explainedabove with reference to FIG. 9C, or as instructed by a teleoperator.

The method of grasping a product item 500 will now be explained withfurther reference to FIG. 15 . If robot 200 has not predetermined thegrasping pose before manipulator robot 200 is in the picking position,the method will begin, at 502, with a command from processor 103 thatinstructs sensors 262 to capture an image of the inventory disposedwithin a target container 110 b. After manipulator robot 200 receivesthe selected grasping pose signal, the robot executes the signal, at504, causing picking arm 206 to perform the selected gasping pose. Thatis, gripping tool 248 approaches the product item, as instructed byprocessor 103, and contacts grasping region 414 of the product item.After the grasping attempt, one of the sensors 262 characterizes thegrasp, at 508, as either successful or unsuccessful. That is, if thepicking arm 206 of robot 200 is able to successfully grasp and removethe product item from target container 110 b, sensor 262 willcharacterize the grasp as successful and transmit a successful graspsignal to processor 103 via communication channels 104. On the otherhand, if the picking arm 206 of robot 200 is unable to remove the itemfrom the container, or the picking arm drops the item before theprocessor 103 instructs robot 200 to release the item, the sensor willcharacterize the grasp as unsuccessful and transmit an unsuccessfulgrasp signal to the processor via communication channels 104. Uponcharacterizing the grasp as unsuccessful, processor 103 can either: (1)immediately signal to teleoperator interface 102, at 510 a, and requestintervention; or (2) attempt to determine a new grasping pose, at 510 b,to autonomously pick up the product item based upon a new or modifiedgrasping pose. If processor 103 elects to autonomously determine a newgrasping pose, the steps described above, with respect to FIG. 9C, maybe repeated until either the grasp is characterized as successful, at512, or until intervention is requested at 510 a.

If processor 103 signals for intervention, the signal may be sentdirectly or indirectly to the teleoperator interface 102. In situationsin which teleoperator interface 102 is communicatively coupled to aplurality of manipulator robots 200, each of the robots may beindirectly coupled to teleoperator interface 102 via a ‘broker’. Thebroker may be part of processor 103, or a separate processor, taskedwith ordering the help requests from each robot within a queue of theteleoperator interface. The broker may run an algorithm to determine a‘needs help score’ to determine the priority of the queue. The algorithmmay be based on several factors including number of prior graspfailures, level of grasping difficulty, and the like.

Once the help request signal has been received by teleoperator interface102, an operator can remotely pilot the picking arm 206 of robot 200 anddirect the picking arm to execute a specified grasping pose to grasp theproduct item. Specifically, the operator can view the items on theoutput device (e.g., the display) of teleoperator interface 102 anddirectly control the picking arm 206 of robot 200 to grasp the graspingregion 414 of the item by manipulating the input device of the operatorinterface. In some instances, the operator may also prompt picking arm206 to grasp a product item in combination with an automated motionsequence calculated by a motion planner. In this manner, the operatormay simply select a pixel on the image feed representative of thegrasping region 414 while processor 103 autonomously determines andinstructs robot 200 to execute a selected grasping pose as describedabove with reference to FIG. 9C.

Sensors 262 can then optionally characterize the grasp as eithersuccessful or unsuccessful as described above at 508. The operator canadditionally, or alternatively, make the same characterization. Ifsensor 262 (or the operator) characterizes the grasp as successful, thegrasping pose used to grasp the product item may be saved within storagedevice 105, at 514, for future use. Robot 200 can thus learn to infer orpredict new grasping poses to improve automation of the graspingprocess.

There is not a single gripping tool that can optimally handle a largevariety of inventory. For this reason, robot 200 may autonomouslydecide, or be instructed from the teleoperator, to switch grippingtools. Gripping tool 248 may be selected based upon the type of task orthe product type (which may be determined by the central computerthrough inventory tacking of the product types in each bin), analysis ofthe image data and/or as a result of historical data relating tosuccessful picks of that product or similar constructed products. Morespecifically, the central computer 103, or an operator, may instructmanipulator robot 200 to couple a particular gripping tool 248 topicking arm 206 that can engage the grasping region 414 of the item withminimal leakage between the gripping tool and the surface of the item.

With reference to FIGS. 9F and 13 , if robot 200 or the teleoperatordetermines that it is desirable to switch gripping tools 248, the robotwill move picking arm 206 to position the groove 249 and/or the sidewall251 of the gripping tool attached to the picking arm within one of theretainers 260 of tool holder 258. Subsequently, the picking arm 206 maybe retracted or moved upward to decouple the magnet 246 of the pickingarm from the magnet 254 of gripping tool 248. The picking arm 206 maythen be positioned over another one of the gripping tools, positionedwithin tool holder 258, to magnetically couple the picking arm to theother one of the gripping tools before moving the picking arm laterallyto slide the coupled gripping tool out of its respective retainers 260.It will be appreciated, however, that other mechanisms for swappinggripping tools may also be utilized.

As gripping tool 248 is brought into contact with the product item, thelip 252 of the gripping tool deforms and conforms to the surface of theproduct as a suction force is applied to grasp the product.Additionally, the compliance in gripping tool 248 and/or picking arm 206will compensate for inaccuracies of the sensing system or graspingalgorithm to position the gripping tool in a better grasping pose uponcontact with the product. With the product grasped, picking arm 206 maythen lift the target product from container 110 and optionally positionor wave/rotate the product in front of scanners 264 to scan anidentifier such as a barcode or RFID located on the target product forthe purpose of confirming that the correct product has been graspedand/or to inform the picking arm as to which order bin 214 the productshould be released. During this time, sensors 262 may additionallycollect data relating to the size and dimension of the product andtransmit this information through communication channels 104 to centralcomputer 103.

In some instances, central computer 103 may then autonomously instruct,or the teleoperator may manually instruct, picking arm 206 to release orplace the grasped item in a particular location and/or orientationwithin order bin 214. Gripping tool 248 and/or tool elements of thegripping tool may then further be used to push, blow on, or otherwisemanipulate the product to a particular location or orientation withinbin 214. In this manner, subsequently picked items may be efficientlypacked within order bin 214 such that smaller order bins may beutilized. This increases the overall amount of order bins that may betransported by a single robot, and in turn, increases the throughput ofthe system. After robot 200 has sequentially picked up each of theproducts corresponding to a particular order, order bins 214 may betransported out of storage structure 114, for example, via shafts 121and the associated conveyor belts, for additional processing, sorting,packaging and/or shipping. If robot 200 is tasked with picking multipleconsumers orders at once, robot 200 need not pick all of the productspertaining to the first consumers order before beginning to pick thesecond consumer's orders. In fact, the central computer will directrobot 200 to pick items based upon the storage locations of the productsand irrespective of the consumer who ordered the product.

In a variant aspect, a manipulator robot is similar to robot 200 but forthe particulars of its pneumatic system as discussed below. Thepneumatic system 300 of the variant robot is schematically illustratedin FIG. 16 . In this variant, the robot does not rely on pneumatic airfrom the storage structure, instead the robot may have a modifiedpneumatic system coupled to the vehicle body of the robot. The pneumaticsystem may include a single or two-tiered vacuum having a first vacuum302 and a second vacuum 304 in selective communication with a grippingtool such as a single suction cup or a modified gripping tool 348 (FIG.17 ). First vacuum 302 may be a vacuum with high flow rate (capable ofdisplacing large volumes of air per minute) while second vacuum 304 maybe strong vacuum generator capable of producing a larger pressuredifferential with atmospheric pressure (which increases the payload orforce that can be held by the suction cup). Pneumatic system 300 furtherincludes two valves 306 a, 306 b (collectively “valves 306”), forexample, servo-valves that may be toggled between an open condition anda closed condition for controlling communication between the first andsecond vacuums and gripping tool 248 or modified gripping tool 348.

As shown in FIG. 17 , modified gripping tool 348 includes a firstsuction cup 308 and a second suction cup 310, which may beconcentrically positioned within the first suction cup. The firstsuction cup 308 and the second suction cup 310 are otherwise formedgenerally as previously described with respect to the suction cup ofgripping tool 248, and therefore, are not again described in detail. Theonly difference being that that modified gripping tool 348 is a dualsuction cup as opposed to the single suction cup of gripping tool 248.First vacuum 302, or the high flow rate vacuum, may be in selectivecommunication with first suction cup 308, while second vacuum, or thehigh pressure vacuum, may be in selective communication with the secondsuction cup 310.

Use of the pneumatic system 300 will now be described only withreference to the grasping task as the variant robot is otherwiseoperated as previously described above with respect to robot 200. Beforegrasping a product, valve 306 a may be transitioned to its openposition, providing the first suction cup 308 with a high flow ratevacuum suction force as the lip of the first suction cup deforms tocorrespondingly match the surface of the target product. After aninitial seal has been initiated, the high pressure vacuum line of vacuum304 is enabled by transitioning valve 306 b to the open condition. Thehigh pressure suction force enables the picking arm to support largerpayloads than would otherwise be possible with the high flow rate vacuumalone. In this manner, a firmer grasp may be provided. Of course, valves306 a and 306 b could both be set to their open positions during initialgrasping of the target product and until the robot desires to releasethe target product. Alternatively, valves 306 a and 306 b could betoggled back and forth and between open, closed and partially closedconditions in order to achieve a desired grasp of the target product.

By utilizing two relatively small vacuum sources, a high flow ratevacuum and a high pressure vacuum, to respectively create an initialseal and to firmly grasp products, the physical size of the vacuums maybe reduced such that the vehicle body of the robot need not be asdramatically modified. In this manner, the variant robot may be used topiece pick products stored within storage structure 114 or within framestructure 14 discussed with respect to the prior art.

FIG. 18 is a perspective view of a modified robotic system 100′configured to efficiently store a plurality of stacked containers 110′.Modified robotic system 100′ includes all of the above describedfeatures of system 100 and the additional features describedhereinafter. Rails 122′, 124′, 125′ may additionally extend above thegrid (supporting a manipulator robot 200 and digging robot 205) andalone, or in combination with additional support members, form a gantryframe that supports one or more robotic picking arms 206′ equipped witha pneumatic gripping tools 248′ in a manner that permits the picking armto move about the gantry frame and piece pick inventory from containers110′. In this manner, compressed air may flow through rails 122′, 124′,125′, and/or the additional rails, to the pneumatic gripping tool 248′of picking arm 206′ for grasping products from containers 110′. Rails122′, 124′, 125′, or the additional support members, positioned abovethe grid may also include one or more valves similar to valve 150, suchthat the valves are accessible to manipulator robot 200 (positioned onthe grid) and allow the manipulator robot to selectively couple to thepneumatic supply system.

Alternatively, robotic picking arms 206′ may be fixed on a frame abovethe grid and digging robot 205 or another bin carrying robot maytransport a target container 110′ to the picking arm, which may graspthe desired item(s) and place the grasped items into an order bincarried by a transporter robot. In this manner, containers 110′ need notbe transported down the ports and back-and-forth from thepicking/sorting stations.

FIG. 19 is a perspective view of yet another alternative robotic system100″ configured to efficiently store a plurality of stacked containers.Robotic system 100″ includes all of the above described features ofrobotic system 100 and the additional features described hereinafter.Manipulator robot 200, or another manipulator robot, may be positionedat a station on grid 126 (e.g., so as to not move) or configured to movewithin a specific area of the grid. These robots may be permanently orselectively coupled to supply lines 140″ that hang from a structureabove the grid, such as the ceiling of warehouse 101, or otherwiseextend toward the surface of grid 126. In some embodiments, supply lines140″ may be retracted, for example, via a drag chain cable carrier,cable retractor or similar device to manage cable slack in the supplylines. Supply lines 140″ may additionally include a power cord, or othermechanism, to supply a voltage to robot 200 when the robot is coupled tothe supply lines. A digging robot such as digging robot 205 cantransport a container 110 to manipulator robot 200, stationed within aparticular area of the grid 200, before inventory is picked from thecontainer and deposited to a plurality of other containers as describedabove with respect robot system 100.

To summarize the foregoing, a storage system for robotic picking, mayinclude support members, a first set of parallel rails to support amobile, manipulator robot and a fluid supply line having a plurality ofvalves disposed within the fluid supply line, and each of the valves mayhave a closed condition in which the fluid supply line is in isolationfrom an outside environment and an open condition in which the fluidsupply line is in communication with the outside environment such thatthe fluid supply line is configured to supply fluid to a mobile,manipulator robot; and/or

-   -   the storage structure may further include a second set of        parallel rails extending substantially perpendicular to the        first set of parallel rails such that the first and second set        of parallel rails form a grid having a plurality of grid spaces;        and/or    -   when the grid is a first grid and the plurality of grid spaces        is a first plurality of grid spaces, the storage system may have        a third set of parallel rails and a fourth set of parallel rails        extending substantially perpendicular to the third set of        parallel rails, such that the third and fourth set of parallel        rails form a second grid elevated above the first grid, whereby        the second grid has a plurality of second grid spaces, and the        storage structure may further include an inclined ramp including        a fifth set of rails connecting the first grid and the second        grid; and/or    -   the storage structure may further include a fluid source in        fluid communication with the fluid supply line, and the fluid        source may be a pneumatic source; and/or    -   the fluid supply line may be attached to an external surface of        the first set of parallel rails; and/or    -   the fluid supply line may be a channel embedded within and        extending in a longitudinal direction of the first set of        parallel rails, and a plurality of conduits may extend between        the channel and a respective port located adjacent a surface of        the first set of parallel rails; and/or    -   at least one of the plurality of valves may be disposed within a        respective one of the conduits; and/or    -   at least one of the plurality of valves may include a biasing        member coupled to a plug, and when the valve is in the closed        condition, the biasing member may biases the plug into the port;        and/or    -   the storage system may further include a mobile, manipulator        robot for picking inventory items stored within the storage        structure, and the robot may include a mobility assembly coupled        to the body and configured to guide movement of the robot along        the first set of parallel rails, a pneumatic coupler sized and        configured to mate with at least one of the plurality of valves        and receive fluid from the supply line, and a picking arm        equipped with a pneumatic gripping tool to pick inventory items;        and/or    -   the first set of parallel rails may include a conductive metal        configured to receive a voltage from a charged or grounded        source; and/or    -   a portion of a surface of the conductive metal may be anodized        or otherwise coated to prevent transfer of the voltage through        the coated surface.

In another aspect, a mobile, manipulator robot is provided forretrieving inventory from a storage structure, and may include a bodyhaving an interface configured to send processor readable data to acentral processor and receive processor executable instructions from thecentral processor, a mobility assembly coupled to the body, a couplerselectively mateable to a port to receive a fluid supply from a supplyline and a picking arm connected to the body, such that the picking armmay be coupled to a first pneumatic gripping tool and configured to pickinventory items; and/or

-   -   the robot may further include a tool holder attached to the        body, and the tool holder may have a plurality of retainers;        and/or    -   the robot may further include a second pneumatic gripping tool        having a different size or material than the first pneumatic        gripping tool, and each of the first and second pneumatic        gripping tools may be are interchangeably coupleable to the        picking arm and disposable within a respective one of the        plurality of retainers; and/or    -   the first pneumatic gripping tool may have an additional tool        element; and/or    -   the robot may include a Venturi pump provided downstream of the        coupler and upstream of the first pneumatic gripping tool;        and/or    -   the robot may include a conductive brush to receive voltage from        a surface contacting the mobility assembly; and/or    -   the mobility assembly may include a plurality of wheels, a motor        and a transmission operably coupling the motor to each of the        plurality of wheels, and the motor may be arranged to control an        orientation of each of the wheels, whereby the wheels are        simultaneously rotatable between a first orientation and a        second orientation; and/or    -   the robot may include a sensor to collect product information        relating to at least one of a surface geometry, surface texture        or porosity from which a grasping region of the inventory items        can be determined; and/or    -   the first pneumatic gripping tool may be a suction cup; and/or    -   the robot may include an air tank coupled to the body, and the        air tank may be less than 20 cubic feet; and/or    -   the inventory may be stored within a container having a height,        and the picking arm may have an end-effector stroke in a        vertical direction that is at least two times the height of the        container; and/or    -   the picking arm may include a base member coupled to the body, a        horizontal extension coupled to the base member, a vertical        extension coupled to the horizontal extension, and a positioning        arm coupled to the vertical member such that the positioning arm        may be translatable relative to the vertical extension, and at        least one of a spring, a back-drivable actuator, a force        controlled actuator, or a compliant gripping element may be        coupled to the positioning arm to provide passive or active        compliance.

In yet another embodiment, a method of controlling a mobile, manipulatorrobot to retrieve a product stored within a container located in astorage structure is disclosed, and the method may include moving themobile, manipulator robot over a first set of parallel rails of thestorage structure and to a picking location, identifying a graspingregion located on a product based at least in part upon image dataobtained by a sensor attached to the mobile, manipulator robot,adjusting a picking arm equipped with a pneumatic gripping tool to agrasping pose, and grasping the product using the pneumatic grippingtool; and/or

-   -   the moving step may be autonomous and at least in part        controlled by a central computer; and/or the adjusting step may        be autonomous and the picking arm may be set to the grasping        pose by a computer configured to predict the grasping pose using        an algorithm; and/or    -   the method may further include determining the grasping pose        during the moving step; and/or    -   the grasping pose or the identification of the grasping region        may be at least in part manually determined by a teleoperator;        and/or    -   the method may further include engaging a coupler of the mobile,        manipulator robot to a port to transfer pneumatic air from a        supply line to the mobile, manipulator robot; and/or    -   the method may further include switching the pneumatic gripping        tool for a different pneumatic gripping tool; and/or    -   the pneumatic gripping tool and the different pneumatic gripping        tool may have different sizes or materials; and/or    -   the method may further include placing the product in an order        container carried by the mobile, manipulator robot or a        different robot; and/or    -   the product may be placed in the order container in an        orientation determined by a computer or a teleoperator        predetermined orientation; and/or    -   the method may further include moving a digging robot to a first        position located over a first stack including a target container        containing the product, extending a digger beneath the body of        the digger robot, securing the target container and one or more        non-target containers positioned over the target container to        the digger, lifting the target container and the one or more        non-target container, moving the digging robot to a second        position located over a second stack of containers missing a        single container, and releasing the target container on top of        the second stack.

Although the disclosure herein has been described with reference toparticular embodiments, it is to be understood that these embodimentsare merely illustrative of the principles and applications of thepresent disclosure. It is therefore to be understood that numerousmodifications may be made to the illustrative embodiments and that otherarrangements may be devised without departing from the spirit and scopeof the present disclosure as defined by the appended claims.

1. A robotic system, comprising: a storage structure, comprising: aframe including vertical members supporting a grid having a first set ofparallel rails extending in a first direction and a second set ofparallel rails extending in a second direction substantiallyperpendicular to the first direction such that the first and second setof parallel rails collectively define grid spaces; and a plurality ofstorage containers arranged in vertical stacks for storing inventoryitems, each one of the vertical stacks configured to be arranged withina respective one of the grid spaces; a load handling robot installedupon the grid to move along the first and second set of parallel rails,the load handling robot including a lifting device extendable in avertical direction to extract one or more storage containers from thevertical stacks and transport the extracted storage container to astation; and a manipulator robot installed at the station and within asingle grid space, the manipulator robot including a picking armprovided with a pneumatic gripping tool arranged to pick an inventoryitem from the transported storage container and place the picked iteminto an order container.
 2. The robotic system of claim 1, furthercomprising a pneumatic supply line coupling the manipulator robot to anexternal pneumatic source.
 3. The robotic system of claim 2, wherein thepneumatic supply line extends at least partially through the storagestructure.
 4. The robotic system of claim 2, wherein the pneumaticsupply line extends at least partially in the vertical direction towardthe grid and the pneumatic supply line is at least partially external ofthe frame.
 5. The robotic system of claim 1, further comprising animaging sensor to capture one or more images of the inventory item. 6.The robotic system of claim 5, wherein the manipulator robot furthercomprises: an interface configured to send processor readable data to acentral computing system and receive processor executable instructionfrom the central computing system; and an onboard processor incommunication with at least one of the imaging sensor or the pickingarm.
 7. The robotic system of claim 6, further comprising a tool holderprovided on the manipulator robot or coupled to the frame, the toolholder being arranged to hold another pneumatic gripping tool.
 8. Therobotic system of claim 7, further comprising a teleoperator interfaceincluding one or more input device configured to receive one or moreinputs from an operator to assist the manipulator robot in picking theinventory item from the transported storage container and/or placing theinventory item in a selected orientation within the order container. 9.The robotic system of claim 8, wherein the picking arm is configured tointerchangeably couple to the pneumatic gripping tool and the anotherpneumatic gripping tool upon receiving instructions from one of thecentral computing system, the onboard processor, or the teleoperatorinterface.
 10. The robotic system of claim 1, wherein the pneumaticgripping tool comprises a plurality of gripping elements.
 11. A roboticsystem, comprising: a storage structure, comprising: a frame includingvertical members supporting a grid having a first set of parallel railsextending in a first direction and a second set of parallel railsextending in a second direction substantially perpendicular to the firstdirection such that the first and second set of parallel railscollectively define grid spaces; and a plurality of storage containersarranged in vertical stacks for storing inventory items, each one of thevertical stacks configured to be arranged within a respective one of thegrid spaces; one or more load handling robots installed upon the gridand arranged to move along the first and second set of parallel rails,each of the load handling robots including a lifting device extendablein a vertical direction to extract one or more storage containers fromthe vertical stacks and transport the extracted storage container to astation; one or more manipulator robots installed at the station on thegrid, each of the manipulator robots including a picking arm providedwith a gripping tool arranged to pick an inventory item from thetransported storage container and place the picked item into an ordercontainer; an imaging sensor to capture one or more images of theinventory item within the transported storage container; a centralcomputing system for coordinating operation of the one or more loadhandling robots about the grid and transmitting manipulation taskinstructions to the one or more manipulator robots; and a teleoperatorinterface configured to: receive the one or more images, theteleoperator interface including one or more input devices configured toreceive one or more inputs from an operator; and transmit, based on theone or more inputs, instructions to the central computing system forassisting the manipulator robot in performing the manipulation taskinstructions, the manipulation task instructions including picking theinventory item from the transported storage container and/or placing theinventory item in a selected orientation within the order container. 12.The robotic system of claim 11, wherein each one of the manipulatorrobots is disposed within a respective single grid space.
 13. Therobotic system of claim 11, further comprising a pneumatic supply linecoupling at least one of the manipulator robots to an external pneumaticsource, wherein the pneumatic supply line extends at least partiallythrough the storage structure.
 14. The robotic system of claim 11,wherein the gripping tool comprises a plurality of gripping elements.15. The robotic system of claim 11, further comprising a tool holderprovided on at least one of the manipulator robot or coupled to theframe, the tool holder being arranged to hold another gripping tool. 16.The robotic system of claim 15, wherein the manipulation taskinstructions include at least one of a partial pose for the grippingtool, a grasping region of the inventory item, or a selection of thegripping tool or the another gripping tool.
 17. A method for assisting amanipulator robot installed at a station on a grid-based storagestructure with a manipulation task, the method comprising: receiving, bya central computing system, data corresponding to the manipulation task,the manipulation task including picking an inventory item from atransported storage container; transmitting, by the central computingsystem to a teleoperator interface, and after receiving the data, one ormore images of the inventory item within the storage container disposedwithin the grid-based storage structure, for assisting the manipulatorrobot with the manipulation task; receiving, by the central computingsystem after transmitting the one or more images, instructions from theteleoperator interface, the instructions including at least one of apartial pose for a gripping tool of the manipulator robot, an identifiedgrasping region of the inventory item, or a selection of a gripping toolto be used by the manipulator robot in performing the manipulation task;and transmitting, by the central computing system, manipulation taskinstructions to the manipulator robot for performing the manipulationtask.
 18. The method of claim 17, wherein the manipulation task furtherincludes placing the inventory item in a selected orientation within anorder container.
 19. The method of claim 17, further comprising:coordinating, by the central computing system, operation of one or moreload handling robots installed upon the grid-based storage structure,the operation of the one or more load handling devices includingmovement along a grid of the grid-based storage structure and/orextraction of the storage container and transportation of the extractedstorage container to the station.
 20. The method of claim 17, whereinthe data comprises product information, grasp success rate incurred whenattempting to grasp that respective inventory item, or a level ofdifficulty in grasping the respective inventory item.